{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "AutoTrace_en_1.03.06.ipynb",
      "provenance": [],
      "collapsed_sections": [],
      "toc_visible": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "accelerator": "GPU"
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "_k0iM-tgybse",
        "colab_type": "text"
      },
      "source": [
        "# Welcome to colab's MMD Auto Trace! (Execution)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "-4KEzyOJQADK",
        "colab_type": "text"
      },
      "source": [
        "# At the beginning\n",
        "\n",
        "The operation status and maintenance information of this tool are posted on Twitter ([@miu200521358](https://twitter.com/miu200521358)).\n",
        "\n",
        "If you get an error, do not start, etc., first check the current distribution status.\n",
        "\n",
        "I also accept inquiries via reply or DM."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "NxA42Uase5hk",
        "colab_type": "text"
      },
      "source": [
        "# MMD Auto Trace Kit preparation"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "aCQg_tD0e9v0",
        "colab_type": "text"
      },
      "source": [
        "This notebook prepares and executes MMD Auto Trace.\n",
        "\n",
        "Click \">\" on the top left of the screen. The table of contents opens.\n",
        "\n",
        "![目次](https://drive.google.com/uc?export=view&id=1x8AdFNmsIQPrtYptBf_NXPRNBJF8ON8z)\n",
        "\n",
        "Check the notebooks from top to bottom and perform the following steps one by one.\n",
        "\n",
        "- **「Environmental setting」**\n",
        "  - Make sure the runtime has changed to GPU\n",
        "   - Please check the introductory section for how to change\n",
        "  - Change Tensorflow version to 1.x\n",
        "  - Download sound effects from [Sound Effect Lab](https://soundeffect-lab.info)\n",
        "   - Sounds during long processing such as preparation and actual trace processing\n",
        "   - Mute your browser volume if you don't need it\n",
        "- **「Cooperation with Google Drive」**\n",
        "  - Make sure you can work with Google Drive\n",
        "  - Please check the introductory version for how to cooperate\n",
        "- **「Preparation batch execution」**\n",
        "    - Run all the cells in the preparation section\n",
        "      - In this process, all programs and data required for MMD Auto Trace are created on colab.\n",
        "      - It takes about 40 to 60 minutes.\n",
        "- **「MMD Auto Trace Kit execution」**\n",
        "  - Execute cells in the execution section one by one from the top\n",
        "    - Specifying Trace Source Video\n",
        "    - Trace parameter settings\n",
        "    - Trace processing execution\n",
        "    - Depending on the number of people, it will take approximately 50 to 60 minutes with 6000 frames.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "F8qp5VzAWyGl",
        "colab_type": "text"
      },
      "source": [
        "##Environmental setting"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "UNrlmW0-W1D1",
        "colab_type": "text"
      },
      "source": [
        "Select \"Runtime\"> \"Change Runtime Type\"> \"GPU\" in the header.\n",
        "\n",
        "If you can change it, please execute the cell below."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "P3zDmyRDwRs0",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "! nvcc --version\n",
        "! nvidia-smi\n",
        "\n",
        "%tensorflow_version 1.x\n",
        "\n",
        "%tensorflow_version\n",
        "\n",
        "! wget -c \"https://soundeffect-lab.info/sound/anime/mp3/sceneswitch1.mp3\"\n",
        "\n",
        "from IPython.display import Audio\n",
        "Audio(\"sceneswitch1.mp3\", autoplay=True)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "oo-BfaAdw9cN",
        "colab_type": "text"
      },
      "source": [
        "**【OK】**\n",
        "\n",
        "It is a success if it is displayed as follows.\n",
        "\n",
        "![GPU変更成功](https://drive.google.com/uc?export=view&id=17CG697kiTLkwVOdH1wg2W3MSB-hyi9u5)\n",
        "\n",
        "---\n",
        "\n",
        "**【NG】**\n",
        "\n",
        "If it is displayed as shown below, the runtime change has failed, so check the introductory version again and change the runtime.\n",
        "\n",
        "![GPU切り替え失敗](https://drive.google.com/uc?export=view&id=1tufSuT7ocWxv3HkrmA5kwlemhu0gv6Je)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "o8OpmLpVp4qr",
        "colab_type": "text"
      },
      "source": [
        "## Cooperation with Google Drive"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "FmdrRx_Tp8Bk",
        "colab_type": "text"
      },
      "source": [
        "Works with the Google Drive `autotrace` folder.\n",
        "\n",
        "Please execute the cell below."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "_f1KFUn_qGsU",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "from google.colab import drive\n",
        "import os\n",
        "\n",
        "# Googleドライブマウント\n",
        "drive.mount('/gdrive')\n",
        "\n",
        "# 起点ディレクトリ\n",
        "base_path = \"/gdrive/My Drive/autotrace\"\n",
        "\n",
        "! echo \"Contents of [autotrace] folder -----------\"\n",
        "! ls -l \"$base_path\"\n",
        "! echo \"--------------------\""
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Fj84B5ir3Rxr",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# Googleドライブアクセスライブラリ\n",
        "!pip install -U -q PyDrive\n",
        "\n",
        "from pydrive.auth import GoogleAuth\n",
        "from pydrive.drive import GoogleDrive\n",
        "from google.colab import auth\n",
        "from oauth2client.client import GoogleCredentials\n",
        " \n",
        "auth.authenticate_user()\n",
        "gauth = GoogleAuth()\n",
        "gauth.credentials = GoogleCredentials.get_application_default()\n",
        "drive = GoogleDrive(gauth)\n",
        "\n",
        "! echo Google SDK Success"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "38aL3FiVWvmN",
        "colab_type": "text"
      },
      "source": [
        "## Preparation batch execution"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "6k716wWUxT4F",
        "colab_type": "text"
      },
      "source": [
        "Here, the cells in the preparation section are executed collectively.\n",
        "\n",
        "Select \"Execute MMD Auto Trace Kit\" in the table of contents.\n",
        "\n",
        "![実行選択](https://drive.google.com/uc?export=view&id=17HmndobtAh5ak4NR0g7E4cElkDEml6tw)\n",
        "\n",
        "If you select \"Runtime\"> \"Perform cells before\" in the header, all cells in the preparation section will be executed sequentially.\n",
        "\n",
        "![すべてのセルを実行](https://drive.google.com/uc?export=view&id=1Y6WJ8EHMKmF9X3M_hH9TZE5VHea1swWn)\n",
        "\n",
        "---\n",
        "\n",
        "** 【OK】**\n",
        "\n",
        "The following is output at the bottom of the screen, and it is complete.\n",
        "\n",
        "![処理成功](https://drive.google.com/uc?export=view&id=1D21xezv6QN0RQF5ZU_LR7PRnOk0Dw4Sc)\n",
        "\n",
        "It takes about 40 to 60 minutes.\n",
        "\n",
        "---\n",
        "\n",
        "**【NG】**\n",
        "\n",
        "![処理失敗](https://drive.google.com/uc?export=view&id=1t-immeF3Ji1_GBNatZOG1C07j42de4Rq)\n",
        "\n",
        "It is a failure if \"No such file or directory\" is output on the last line.\n",
        "\n",
        "If you don't know the solution, share your notebook."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "E-h6RWCXnZU8",
        "colab_type": "text"
      },
      "source": [
        "### Environmental setting"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "IPiDvSBanScr",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# 処理時間計測のための開始時間\n",
        "import time\n",
        "start_time = time.time()"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "9UMIfs3snkRm",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# Openposeバージョンタグ\n",
        "ver_openpose = \"v1.5.1\""
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "iWQcbt_rnblK",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# MMD自動トレースキットバージョンタグ\n",
        "ver_tag = \"ver1.03.02\""
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "EK-WUxgciv9V",
        "colab_type": "text"
      },
      "source": [
        "### cmake"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "djxuuJjKix5B",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "! wget -c \"https://github.com/Kitware/CMake/releases/download/v3.17.2/cmake-3.17.2.tar.gz\"\n",
        "! tar xf cmake-3.17.2.tar.gz\n",
        "! cd cmake-3.17.2 && ./configure && make && sudo make install"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Viqw8qJqfDyf",
        "colab_type": "text"
      },
      "source": [
        "### Openpose"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "a-fnE9kwgcfg",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# ライブラリのインストール\n",
        "\n",
        "# Basic\n",
        "! sudo apt-get --assume-yes update\n",
        "! sudo apt-get --assume-yes install build-essential\n",
        "# OpenCV\n",
        "! sudo apt-get --assume-yes install libopencv-dev\n",
        "# General dependencies\n",
        "! sudo apt-get --assume-yes install libatlas-base-dev libprotobuf-dev libleveldb-dev libsnappy-dev libhdf5-serial-dev protobuf-compiler\n",
        "! sudo apt-get --assume-yes install --no-install-recommends libboost-all-dev\n",
        "# Remaining dependencies, 14.04\n",
        "! sudo apt-get --assume-yes install libgflags-dev libgoogle-glog-dev liblmdb-dev\n",
        "# Python2 libs\n",
        "! sudo apt-get --assume-yes install python-setuptools python-dev build-essential\n",
        "! sudo easy_install pip\n",
        "! sudo -H pip install --upgrade numpy protobuf opencv-python\n",
        "# Python3 libs\n",
        "! sudo apt-get --assume-yes install python3-setuptools python3-dev build-essential\n",
        "! sudo apt-get --assume-yes install python3-pip\n",
        "! sudo -H pip3 install --upgrade numpy protobuf opencv-python\n",
        "# OpenCV 2.4 -> Added as option\n",
        "# # sudo apt-get --assume-yes install libopencv-dev\n",
        "# OpenCL Generic\n",
        "! sudo apt-get --assume-yes install opencl-headers ocl-icd-opencl-dev\n",
        "! sudo apt-get --assume-yes install libviennacl-dev"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "eMCKuBuagoFo",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "#  Openpose の clone\n",
        "! git clone  --depth 1 -b \"$ver_openpose\" https://github.com/CMU-Perceptual-Computing-Lab/openpose.git \n",
        "# ! git clone  --depth 1 https://github.com/CMU-Perceptual-Computing-Lab/openpose.git     "
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "4E8Teu7HlNZC",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# OpenposeモデルデータDL\n",
        "! rm -r ./openpose/models\n",
        "\n",
        "# Openposeモデルデータ の導入\n",
        "! mkdir -p ./openpose/models\n",
        "\n",
        "# Openposeモデルデータのダウンロード\n",
        "downloaded = drive.CreateFile({'id': '1-XlC0b00ueiS8-fDqKeS8w5NQ35DOnFN'})\n",
        "downloaded.GetContentFile('opnepose-models.zip')\n",
        "\n",
        "! unzip ./opnepose-models.zip\n",
        "! mv ./models ./openpose"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "BSls4lfOgwG0",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# Create build directory\n",
        "! cd openpose && mkdir build && cd build"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ucMY7PLlZ69S",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# Scenario 1 - Caffe not installed and OpenCV installed using apt-get\n",
        "! cd openpose/build && cmake ..  # -D DOWNLOAD_BODY_COCO_MODEL=ON"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "8ld2HCrvg7c3",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# Openpose BUilding\n",
        "! cd openpose/build && make -j`nproc`"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "lZLayojmhDdI",
        "colab_type": "text"
      },
      "source": [
        "### mannequinchallenge-vmd"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "rJw3lnjKhIp7",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# mannequinchallenge-vmd の clone\n",
        "! git clone  --depth 1 -b \"$ver_tag\" https://github.com/miu200521358/mannequinchallenge-vmd.git"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "sNbLo-sBhlcL",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# mannequinchallenge-vmd の モデルデータDL\n",
        "\n",
        "# モデルデータのダウンロード\n",
        "! cd  ./mannequinchallenge-vmd && ./fetch_checkpoints.sh"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Gvkd3YfCiVJ8",
        "colab_type": "text"
      },
      "source": [
        "### 3d-pose-baseline-vmd"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "w3Uh4e6liYPg",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# 3d-pose-baseline-vmd の clone\n",
        "! git clone  --depth 1 -b \"$ver_tag\" https://github.com/miu200521358/3d-pose-baseline-vmd.git"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Iw7kI9zhi_7k",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# 3d-pose-baseline-vmd の Human3.6MモデルデータDL\n",
        "\n",
        "# Human3.6Mモデルデータ の導入\n",
        "! mkdir -p ./3d-pose-baseline-vmd/data/h36m\n",
        "\n",
        "# Human3.6Mモデルデータのダウンロード\n",
        "downloaded = drive.CreateFile({'id': '1W5WoWpCcJvGm4CHoUhfIB0dgXBDCEHHq'})\n",
        "downloaded.GetContentFile('h36m.zip')\n",
        "\n",
        "! unzip ./h36m.zip\n",
        "\n",
        "! mv ./h36m ./3d-pose-baseline-vmd/data/"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "dclND00zjGdN",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# 3d-pose-baseline-vmd の 学習データDL\n",
        "\n",
        "# 3d-pose-baseline用学習データ の導入\n",
        "! mkdir -p ./3d-pose-baseline-vmd/experiments\n",
        "\n",
        "# Human3.6Mモデルデータのダウンロード\n",
        "downloaded = drive.CreateFile({'id': '1v7ccpms3ZR8ExWWwVfcSpjMsGscDYH7_'})\n",
        "downloaded.GetContentFile('experiments.zip')\n",
        "\n",
        "! unzip ./experiments.zip\n",
        "\n",
        "! mv ./experiments ./3d-pose-baseline-vmd/"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "jruMP1J4jLXX",
        "colab_type": "text"
      },
      "source": [
        "### VMD-3d-pose-baseline-multi"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "87ZPjj6IjPgj",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# VMD-3d-pose-baseline-multi の clone\n",
        "\n",
        "! git clone  --depth 1 -b \"$ver_tag\" https://github.com/miu200521358/VMD-3d-pose-baseline-multi.git"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "fnuSwMT9jW5E",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# VMD-3d-pose-baseline-multi のライブラリ\n",
        "\n",
        "! sudo apt-get install python3-pyqt5  \n",
        "! sudo apt-get install pyqt5-dev-tools\n",
        "! sudo apt-get install qttools5-dev-tools"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "GIp8lIjZY7ih",
        "colab_type": "text"
      },
      "source": [
        "### Preparation result confirmation"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "T8vXn_ZE6bHk",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# サンプルの実行確認\n",
        "! cd openpose && ./build/examples/openpose/openpose.bin --video examples/media/video.avi --write_json ./output/ --display 0  --write_video ./output/openpose.avi"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "gZjjbPz0llYs",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "import time\n",
        "\n",
        "elapsed_time = (time.time() - start_time) / 60\n",
        "\n",
        "! echo \"■■■■■■■■■■■■■■■■■■■■■■■■\"\n",
        "! echo \"■■All processing has been completed\"\n",
        "! echo \"■■\"\n",
        "! echo \"■■Time：\" \"$elapsed_time\" \"\"\n",
        "! echo \"■■■■■■■■■■■■■■■■■■■■■■■■\"\n",
        "\n",
        "! echo \"Openpose Result\"\n",
        "\n",
        "! ls -l ./openpose/output/openpose.avi\n",
        "\n",
        "from IPython.display import Audio\n",
        "Audio(\"sceneswitch1.mp3\", autoplay=True)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "IrnCtUv8XohO",
        "colab_type": "text"
      },
      "source": [
        "# Execute MMD Auto Trace Kit"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "3g59mu0Q5fC8",
        "colab_type": "text"
      },
      "source": [
        "## Installation complete confirmation"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "l5-4naWjXqUo",
        "colab_type": "text"
      },
      "source": [
        "From here, we will actually execute the kit.\n",
        "\n",
        "In \"Run previous cell\", did all the installation have been performed?\n",
        "\n",
        "For more information, check the Getting Started section.\n",
        "\n",
        "Once you're ready, run the cell below to see if the installation is complete."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "soBmKdn_KjW5",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "!ls -l ./openpose/README.md\n",
        "!ls -l ./mannequinchallenge-vmd/README.md\n",
        "!ls -l ./3d-pose-baseline-vmd/README.md\n",
        "!ls -l ./VMD-3d-pose-baseline-multi/README.md\n",
        "!ls -l ./openpose/output/openpose.avi"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Dd1C8N7iL4ZJ",
        "colab_type": "text"
      },
      "source": [
        "**【OK】**\n",
        "\n",
        "If the file name and file size are displayed as shown below, installation is complete.\n",
        "\n",
        "Please proceed to the input video file upload.\n",
        "\n",
        "![インストール成功](https://drive.google.com/uc?export=view&id=1l13A2iF9oTpGcZSe9q8k7JyDyiABxOQT)\n",
        "\n",
        "---\n",
        "\n",
        "**【NG】**\n",
        "\n",
        "If \"No such file or directory\" is displayed as shown below, installation fails.\n",
        "\n",
        "![インストール失敗](https://drive.google.com/uc?export=view&id=1LuKoSMwFOzFg8NguFxqAtmQy9B1_KMXr)\n",
        "\n",
        "Return to the top of this notebook and try again from the beginning.\n",
        "\n",
        "If the installation fails after three attempts, share your notebook."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "4R8ogPNZXtQc",
        "colab_type": "text"
      },
      "source": [
        "## Input video file upload"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "FrglOLBZXv9B",
        "colab_type": "text"
      },
      "source": [
        "Prepare the video file you want to process.\n",
        "\n",
        " - The file name should be **only single-byte alphanumeric characters**. opencv can not read double-byte characters.\n",
        " - Please put it under the **autotrace** folder of Google Drive.\n",
        " - FPS should be **30fps** or **60fps**.\n",
        " - The size should be **1280x720**.\n",
        " - If the size or fps is not as specified, the program will re-encode. (Fps will be 30)\n",
        " - **Overwriting or updating files on the Goole drive after mounting is not recognized correctly.** Please upload a new file with a new name and process it.\n",
        " - When tracing the number of people, if people disappear from the screen, the order can not be acquired and the traces of the unplaced persons will be incorrect. Make sure that all the members of the person you want to acquire are displayed as much as possible.\n",
        " - Once the upload is complete, please execute the cells below sequentially."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "WQxj2Y6-Zutl",
        "colab_type": "code",
        "cellView": "form",
        "colab": {}
      },
      "source": [
        "#@markdown ### 【O】Input video file\n",
        "#@markdown Enter the name of the video file to be analyzed.\n",
        "#@markdown If the width is not 1280 or if the frame rate is not 30 fps, re-encoding is performed.\n",
        "\n",
        "input_video_name = \"input.mp4\"  #@param {type: \"string\"}\n",
        "\n",
        "from google.colab import drive\n",
        "import sys\n",
        "import os\n",
        "import cv2\n",
        "import datetime\n",
        "import math\n",
        "from tqdm import tqdm\n",
        "from PIL import Image\n",
        "import traceback\n",
        "import numpy as np\n",
        "\n",
        "# Googleドライブマウント\n",
        "drive.mount('/gdrive')\n",
        "\n",
        "# 起点ディレクトリ\n",
        "base_path = \"/gdrive/My Drive/autotrace\"\n",
        "\n",
        "! echo \"autotrace folder -----------\"\n",
        "! ls -l \"$base_path\"\n",
        "! echo \"--------------------\"\n",
        "\n",
        "# 入力動画ファイル\n",
        "input_video = base_path + \"/\"+ input_video_name\n",
        "\n",
        "print(\"file name: \", os.path.basename(input_video))\n",
        "print(\"file size: \", os.path.getsize(input_video))\n",
        "\n",
        "\n",
        "video = cv2.VideoCapture(input_video)\n",
        "# 幅\n",
        "W = video.get(cv2.CAP_PROP_FRAME_WIDTH)\n",
        "# 高さ\n",
        "H = video.get(cv2.CAP_PROP_FRAME_HEIGHT)\n",
        "# 総フレーム数\n",
        "count = video.get(cv2.CAP_PROP_FRAME_COUNT)\n",
        "# fps\n",
        "fps = video.get(cv2.CAP_PROP_FPS)\n",
        "\n",
        "print(\"width: {0}, height: {1}, frames: {2}, fps: {3}\".format(W, H, count, fps))\n",
        "\n",
        "\n",
        "\n",
        "if W >= H:\n",
        "    # 横長の場合\n",
        "    width = 1280\n",
        "    height = 720\n",
        "else:\n",
        "    # 縦長の場合\n",
        "    height = 1280\n",
        "    width = 720\n",
        "\n",
        "# 画面比率\n",
        "aspect = width / height\n",
        "\n",
        "if (aspect != (9/6) and aspect != (16/9)) or W != 1280 or H != 720 or (fps != 30 and fps != 60):\n",
        "    print(\"Re-encode because size or fps is not processed: \"+ input_video)\n",
        "    \n",
        "    # 補間png出力先\n",
        "    interpolation_output_path = \"./interpolation.png\"\n",
        "\n",
        "    # 縮尺\n",
        "    scale = width / W\n",
        "\n",
        "    # オリジナル高さ\n",
        "    im_height = int(H * scale)\n",
        "\n",
        "    # 出力ファイルパス\n",
        "    out_name = 'recode_{0}.mp4'.format(\"{0:%Y%m%d_%H%M%S}\".format(datetime.datetime.now()))\n",
        "    out_path = '{0}/{1}'.format(base_path, out_name)\n",
        "    cnt = 0\n",
        "\n",
        "    try:\n",
        "        fourcc = cv2.VideoWriter_fourcc(*\"MP4V\")\n",
        "        out = cv2.VideoWriter(out_path, fourcc, 30.0, (width, height))\n",
        "        # 入力ファイル\n",
        "        cap = cv2.VideoCapture(input_video)\n",
        "        # オリジナル出力フレーム\n",
        "        output_frames = []\n",
        "\n",
        "        print (\"Start importing original video\")\n",
        "\n",
        "        for _ in range(int(count)):\n",
        "        # for _ in tqdm(range(int(count))):\n",
        "            if not cap.isOpened():\n",
        "                break\n",
        "\n",
        "            # 動画から1枚キャプチャして読み込む\n",
        "            flag, frame = cap.read()  # Capture frame-by-frame\n",
        "\n",
        "            # 動画が終わっていたら終了\n",
        "            if flag == False:\n",
        "                break\n",
        "\n",
        "            # 画像の縦横を指定サイズに変形\n",
        "            img = Image.fromarray(frame)\n",
        "            img = img.resize((width, im_height),Image.ANTIALIAS)\n",
        "\n",
        "            # 黒く塗りつぶす用の背景画像を作成\n",
        "            bg = Image.new(\"RGB\",[width,height],(0,0,0))\n",
        "\n",
        "            # 元の画像を、背景画像のセンターに配置\n",
        "            bg.paste(img,(int((width-img.size[0])/2),int((height-img.size[1])/2)))\n",
        "\n",
        "            # opencv用に変換\n",
        "            out_frame = np.asarray(bg)\n",
        "\n",
        "            output_frames.append(out_frame)\n",
        "\n",
        "        print (\"Finished loading the original video\")\n",
        "\n",
        "        # 補間後のフレーム\n",
        "        interpolarted_frames = []\n",
        "        # フレーム補間用比率\n",
        "        fps_interpolation = fps / 30\n",
        "\n",
        "        print(\"Start interpolation video generation\")\n",
        "        cnt = 0\n",
        "\n",
        "        # 最後の１つ手前（補間ができる状態）までループ\n",
        "        # for _ in tqdm(range(round(count * (30 / fps)) - 1)):\n",
        "        for _ in range(round(count * (30 / fps)) - 1):\n",
        "            # 補間した出力CNT\n",
        "            inter_cnt = cnt * fps_interpolation\n",
        "            # INDEXと比率（整数部と小数部）\n",
        "            inter_cnt_rate, inter_cnt_idx = math.modf(inter_cnt)\n",
        "            # print(\"フレーム補間: %s -> %s, idx: %s, rate: %s\" % ( cnt, inter_cnt, inter_cnt_idx, inter_cnt_rate ))\n",
        "\n",
        "            # 前のフレーム\n",
        "            past_frame = output_frames[int(inter_cnt_idx)]\n",
        "            # 今回のフレーム\n",
        "            now_frame = output_frames[int(inter_cnt_idx + 1)]\n",
        "\n",
        "            # 混ぜ合わせる比率\n",
        "            past_rate = inter_cnt_rate\n",
        "            now_rate = 1 - inter_cnt_rate\n",
        "\n",
        "            # フレーム補間をして出力する\n",
        "            target_output_frame = cv2.addWeighted(past_frame, past_rate, now_frame, now_rate, 0)\n",
        "\n",
        "            # # PNG出力\n",
        "            # cv2.imwrite(interpolation_output_path, target_output_frame.clip(0, 255))\n",
        "\n",
        "            # # 再読み込み\n",
        "            # interpolation_img = cv2.imread(interpolation_output_path)\n",
        "\n",
        "            # 動画出力\n",
        "            out.write(target_output_frame)\n",
        "\n",
        "            cnt += 1\n",
        "\n",
        "        print(\"Completion of interpolation video generation\")\n",
        "\n",
        "        # 最後にnowを出力\n",
        "        out.write(now_frame)\n",
        "\n",
        "        # 終わったら開放\n",
        "        out.release()\n",
        "        cap.release()\n",
        "    except Exception as e:\n",
        "        print(\"Re-encoding failure\", e)\n",
        "        print(traceback.format_exc())\n",
        "\n",
        "    cv2.destroyAllWindows()\n",
        "    \n",
        "    print('Regenerate MP4 file for MMD input: ', out_path)\n",
        "    input_video_name = out_name\n",
        "\n",
        "    # 入力動画ファイル再設定\n",
        "    input_video = base_path + \"/\"+ input_video_name\n",
        "    \n",
        "    video = cv2.VideoCapture(input_video)\n",
        "    # 幅\n",
        "    W = video.get(cv2.CAP_PROP_FRAME_WIDTH)\n",
        "    # 高さ\n",
        "    H = video.get(cv2.CAP_PROP_FRAME_HEIGHT)\n",
        "    # 総フレーム数\n",
        "    count = video.get(cv2.CAP_PROP_FRAME_COUNT)\n",
        "    # fps\n",
        "    fps = video.get(cv2.CAP_PROP_FPS)\n",
        "\n",
        "    print(\"[Re-check] width: {0}, height: {1}, frames: {2}, fps: {3}\".format(W, H, count, fps))\n",
        "\n",
        "    \n",
        "!echo \"The input video is\" \"$input_video_name\" \".\"\n",
        "!echo \"\"\n",
        "!echo \"If there is no problem, proceed to the next.\"\n",
        "\n",
        "from IPython.display import Audio\n",
        "Audio(\"sceneswitch1.mp3\", autoplay=True)\n"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "LXcgFDk-YDMB",
        "colab_type": "text"
      },
      "source": [
        "It is successful if the file name can be obtained at the end of the process.\n",
        "\n",
        "---\n",
        "**【OK】**\n",
        "\n",
        "If the file size and fps are as specified, the input file is handled as it is.\n",
        "\n",
        "![OK](https://drive.google.com/uc?export=view&id=1lvOhCAj99_NUNDb-wfRAxeu-o0Exth7v)\n",
        "\n",
        "----\n",
        "**【Re-encoding】**\n",
        "\n",
        "If the file size and fps are not as specified, the re-encoded mp4 file is treated as an input file.\n",
        "\n",
        "![再エンコード](https://drive.google.com/uc?export=view&id=1xEiy-pdeHWQpt4CLZePg9YT1_7U8bEqz)\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "8qMeT79QYFeC",
        "colab_type": "text"
      },
      "source": [
        "## Parameter setting"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "AFlJGTLmFLXE",
        "colab_type": "text"
      },
      "source": [
        "Please set the parameters.\n",
        "\n",
        " - 【O】… Parameters used by Openpose\n",
        " - 【M】… Parameters used by mannequinchallenge-vmd\n",
        " - 【V】… Parameters used by VMD-3d-pose-baseline-multi"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "FQ3yCAl8YI1o",
        "colab_type": "code",
        "cellView": "form",
        "colab": {}
      },
      "source": [
        "#@markdown Enter the parameters for tracing the image and execute the cell.\n",
        "\n",
        "#@markdown --- \n",
        "\n",
        "#@markdown ### 【O】Maximum number of people in the video\n",
        "#@markdown Please enter the number of people you want to get from the video.\n",
        "#@markdown Please process the video data as much as possible for this number of people.\n",
        "number_people_max = 1  #@param {type: \"number\"}\n",
        "\n",
        "#@markdown --- \n",
        "\n",
        "#@markdown ### 【O】Frame number to start analysis\n",
        "#@markdown Enter the frame number to start analysis. (0 beginning)\n",
        "#@markdown If the human body can not be traced accurately, for example, if the logo is displayed first, specify the first frame in which all the members appear in the image.\n",
        "frame_first = 0  #@param {type: \"number\"}\n",
        "\n",
        "#@markdown ---\n",
        "\n",
        "#@markdown ### 【M】Frame number to finish analysis\n",
        "#@markdown Please enter the frame number to finish the analysis. (0 beginning)\n",
        "#@markdown When you adjust the reverse or order in \"FCRN-DepthPrediction-vmd\", you can finish the process and see the result without outputting to the end.\n",
        "#@markdown If the default value is \"-1\", analysis is performed to the end.\n",
        "end_frame_no = -1  #@param {type: \"number\"}\n",
        "\n",
        "#@markdown --- \n",
        "\n",
        "#@markdown ### 【M】Reverse specification list\n",
        "#@markdown Specify the frame number (0 starting) that is inverted by Openpose by mistake, the person INDEX order, and the contents of the inversion.\n",
        "#@markdown In the order that Openpose recognizes at 0F, INDEX is assigned as 0, 1, ....\n",
        "#@markdown Format: [{frame number}: Person who wants to specify reverse INDEX, {reverse content}]\n",
        "#@markdown {reverse content}: R: Whole body inversion, U: Upper body inversion, L: Lower body inversion, N: No inversion\n",
        "#@markdown ex）[10:1,R]　…　The whole person flips the first person in the 10th frame.\n",
        "#@markdown Since the contents are output in the above format in message.log when inverted output, please refer to that.\n",
        "#@markdown As in [10:1,R][30:0,U], multiple items can be specified in parentheses.\n",
        "reverse_specific = \"\"  #@param {type: \"string\"}\n",
        "\n",
        "#@markdown --- \n",
        "\n",
        "#@markdown ### 【M】Ordered list\n",
        "#@markdown In the multi-person trace, please specify the person INDEX order after crossing.\n",
        "#@markdown In the case of a one-person trace, it is OK to leave it blank.\n",
        "#@markdown In the order that Openpose recognizes at 0F, INDEX is assigned as 0, 1, ....\n",
        "#@markdown Format: [{frame number}: index of first estimated person, index of first estimated person, ...]\n",
        "#@markdown 例）[10:1,0]　…　The order of the 10th frame is rearranged in the order of the first person from the left and the zeroth person.\n",
        "#@markdown The order in which messages are output in message.log is left in the above format, so please refer to it.\n",
        "#@markdown As in [10:1,0][30:0,1], multiple items can be specified in parentheses.\n",
        "#@markdown Also, in output_XXX.avi, colors are assigned to people in the estimated order. The right half of the body is red and the left half is the following color.\n",
        "#@markdown 0: green, 1: blue, 2: white, 3: yellow, 4: peach, 5: light blue, 6: dark green, 7: dark blue, 8: gray, 9: dark yellow, 10: dark peach, 11: dark light blue\n",
        "order_specific = \"\"  #@param {type: \"string\"}\n",
        "\n",
        "#@markdown --- \n",
        "\n",
        "#@markdown ### 【V】Bone structure CSV file\n",
        "#@markdown Select or enter the path of the bone structure CSV file of the trace target model.\n",
        "#@markdown You can select \"Animasa-Miku\" and \"Animasa-Miku semi-standard\", or you can input a bone structure CSV file of any model.\n",
        "#@markdown If you want to input any model bone structure CSV file, please upload the csv file to the \"autotrace\" folder of Google Drive.\n",
        "#@markdown And please enter like「/gdrive/My Drive/autotrace/[csv file name]」\n",
        "born_model_csv = \"born/animasa_miku_born.csv\" #@param [\"born/animasa_miku_born.csv\", \"born/animasa_miku_semi_standard_born.csv\"] {allow-input: true}\n",
        "\n",
        "\n",
        "#@markdown --- \n",
        "\n",
        "#@markdown ### 【V】Whether to output with IK\n",
        "#@markdown Output the foot of trace data as IK, or select yes or no.\n",
        "#@markdown If you enter no, output with FK\n",
        "ik_flag = \"yes\"  #@param ['yes', 'no']\n",
        "is_ik = 1 if ik_flag == \"yes\" else 0\n",
        "\n",
        "#@markdown ---\n",
        "\n",
        "#@markdown ### 【V】Heel position correction\n",
        "#@markdown Please input the Y axis correction value of the heel with a numerical value (decimal possible).\n",
        "#@markdown Entering a negative value approaches the ground, entering a positive value moves away from the ground.\n",
        "#@markdown Although it corrects automatically to some extent automatically, if you can not correct it, please set it.\n",
        "heel_position = 0.0  #@param {type: \"number\"}\n",
        "\n",
        "#@markdown ---\n",
        "\n",
        "#@markdown ### 【V】Center-Z moving magnification\n",
        "#@markdown Please enter the magnification multiplied by the center Z movement with a numerical value (decimal possible).\n",
        "#@markdown The smaller the value, the smaller the width of the center Z movement.\n",
        "#@markdown When 0 is input, center Z axis movement is not performed.\n",
        "center_z_scale = 1.5  #@param {type: \"number\"}\n",
        "\n",
        "#@markdown ---\n",
        "\n",
        "#@markdown ### 【V】Center-Z Smoothing frequency\n",
        "#@markdown Specify the degree of motion smoothing center-z\n",
        "#@markdown Please enter only an integer of 1 or more.\n",
        "#@markdown The larger the frequency, the smoother it is. (The behavior will be smaller instead)\n",
        "depth_smooth_times = 4  #@param {type: \"number\"}\n",
        "\n",
        "#@markdown ---\n",
        "\n",
        "#@markdown ### 【V】Smoothing frequency\n",
        "#@markdown Specify the degree of motion smoothing\n",
        "#@markdown Please enter only an integer of 1 or more.\n",
        "#@markdown The larger the frequency, the smoother it is. (The behavior will be smaller instead)\n",
        "smooth_times = 1  #@param {type: \"number\"}\n",
        "\n",
        "#@markdown ---\n",
        "\n",
        "#@markdown ### 【V】Movement key thinning amount\n",
        "#@markdown Specify the amount of movement to be used for decimation of movement key (IK, center) with numerical value (decimal possible)\n",
        "#@markdown When there is a movement within the specified range, it is thinned out.\n",
        "#@markdown When moving thinning amount is set to 0, thinning is not performed.\n",
        "threshold_pos = 0.5  #@param {type: \"number\"}\n",
        "\n",
        "#@markdown ---\n",
        "\n",
        "#@markdown ### 【V】Rotating Key Culling Angle\n",
        "#@markdown Specify the angle (decimal possible from 0 to 180 degrees) to be used for decimating rotation keys\n",
        "#@markdown It will be thinned out if there is a rotation within the specified angle.\n",
        "threshold_rot = 3  #@param {type: \"number\"}\n",
        "\n",
        "\n",
        "import os\n",
        "if not os.path.exists(\"./VMD-3d-pose-baseline-multi/{0}\".format(born_model_csv)):\n",
        "    # 既存のボーン構造CSVでない場合、ドライブの下を参照\n",
        "    born_model_csv = \"/gdrive/My Drive/autotrace/{0}\".format(born_model_csv)\n",
        "    if not os.path.exists(born_model_csv):\n",
        "        !echo ■■■■■■■■■■■■■■■■\n",
        "        !echo ■ WARNING\n",
        "        !echo ■ Bone structure CSV not found. Check the file name.\n",
        "        !echo ■ \"$born_model_csv\"\n",
        "        !echo ■■■■■■■■■■■■■■■■\n",
        "\n",
        "\n",
        "!echo 【O】Maximum number of people in the video: \"$number_people_max\"\n",
        "!echo 【O】Frame number to start analysis: \"$frame_first\"\n",
        "!echo 【F】Frame number to finish analysis: \"$end_frame_no\"\n",
        "!echo 【F】Reverse specification list: \"$reverse_specific\"\n",
        "!echo 【F】Ordered list: \"$order_specific\"\n",
        "!echo 【V】Bone structure CSV file: \"$born_model_csv\"\n",
        "!echo 【V】Whether to output with IK: \"$ik_flag\"\n",
        "!echo 【V】Heel position correction: \"$heel_position\"\n",
        "!echo 【V】Center Z moving magnification: \"$center_z_scale\"\n",
        "!echo 【V】Smoothing frequency: \"$smooth_times\"\n",
        "!echo 【V】Movement key thinning amount: \"$threshold_pos\"\n",
        "!echo 【V】Rotating Key Culling Angle: \"$threshold_rot\"\n",
        "\n",
        "!echo \"\"\n",
        "!echo If the above is correct, please proceed to the next."
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "SFWBO6c0YLa4",
        "colab_type": "text"
      },
      "source": [
        "## Auto Trace execution (all execution)"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Zz0bFW4CYQJ9",
        "colab_type": "code",
        "cellView": "form",
        "colab": {}
      },
      "source": [
        "#@markdown Please execute this cell after completing all the forms.\n",
        "#@markdown Processing is performed in the following order.\n",
        "\n",
        "#@markdown 1. Openpose（Video→2D）\n",
        "#@markdown 2. mannequinchallenge-vmd（Depth estimation）\n",
        "#@markdown 3. 3d-pose-baseline-vmd（2D→3D）\n",
        "#@markdown 4. VMD-3d-pose-baseline-multi（3D→VMD）\n",
        "\n",
        "#@markdown Depending on the number of traces, it takes about 50 to 60 minutes in 6000 frames.\n",
        "#@markdown When Openpose starts, it looks like it has stopped moving for a while with its elongated square.\n",
        "#@markdown If the playback button is spinning around, processing is taking place behind the scenes, so please wait without doing anything.\n",
        "#@markdown If the vmd file has not been generated, the contents of pos.txt are empty, there is only error.txt, etc., first check the contents of error.txt, and check and execute the \"If an error occurs\" section please do it.\n",
        "\n",
        "import time\n",
        "import datetime\n",
        "import cv2\n",
        "import shutil\n",
        "import glob\n",
        "from google.colab import drive\n",
        "\n",
        "start_time = time.time()\n",
        "\n",
        "# 出力フォルダ削除\n",
        "if os.path.exists(\"./output\"):\n",
        "    !rm -r ./output\n",
        "\n",
        "# 処理日時\n",
        "now_str = \"{0:%Y%m%d_%H%M%S}\".format(datetime.datetime.now())\n",
        "\n",
        "# Googleドライブマウント\n",
        "drive.mount('/gdrive')\n",
        "\n",
        "# 起点ディレクトリ\n",
        "drive_base_dir = \"/gdrive/My Drive/autotrace\"\n",
        "\n",
        "output_json = \"/content/output/json\"\n",
        "output_openpose_avi = \"/content/output/openpose.avi\"\n",
        "! mkdir -p \"$output_json\"\n",
        "\n",
        "# 出力用Googleドライブフォルダ\n",
        "drive_dir_path = drive_base_dir + \"/\" + now_str \n",
        "! mkdir -p \"$drive_dir_path\"\n",
        "\n",
        "! echo ------------------------------------------\n",
        "! echo Openpose\n",
        "! echo ------------------------------------------\n",
        "\n",
        "# Openpose実行\n",
        "! cd openpose/ && ./build/examples/openpose/openpose.bin --video \"$input_video\" --display 0 --model_pose COCO --write_json \"$output_json\" --write_video \"$output_openpose_avi\" --frame_first \"$frame_first\" --number_people_max \"$number_people_max\"\n",
        "\n",
        "! echo ------------------------------------------\n",
        "! echo mannequinchallenge-vmd\n",
        "! echo ------------------------------------------\n",
        "\n",
        "! cd mannequinchallenge-vmd && python predict_video.py --video_path \"$input_video\" --json_path \"$output_json\" --interval 20 --reverse_specific \"$reverse_specific\" --order_specific \"$order_specific\" --verbose 1 --now \"$now_str\" --avi_output \"yes\"  --number_people_max \"$number_people_max\" --end_frame_no \"$end_frame_no\" --input single_view --batchSize 1\n",
        "    \n",
        "# 深度結果コピー\n",
        "depth_dir_path =  output_json + \"_\" + now_str + \"_depth\"\n",
        "\n",
        "if os.path.exists( depth_dir_path + \"/error.txt\"):\n",
        "    \n",
        "    # エラー発生\n",
        "    ! cp \"$depth_dir_path\"/error.txt \"$drive_dir_path\"\n",
        "\n",
        "    ! echo \"■■■■■■■■■■■■■■■■■■■■■■■■\"\n",
        "    ! echo \"■■Processing was interrupted due to an error.\"\n",
        "    ! echo \"■■\"\n",
        "    ! echo \"■■■■■■■■■■■■■■■■■■■■■■■■\"\n",
        "\n",
        "    ! echo \"$drive_dir_path\" \" - Check the contents of error.txt.\"\n",
        "\n",
        "else:\n",
        "    \n",
        "    ! cp \"$depth_dir_path\"/*.avi \"$drive_dir_path\"\n",
        "    ! cp \"$depth_dir_path\"/message.log \"$drive_dir_path\"\n",
        "    ! cp \"$depth_dir_path\"/reverse_specific.txt \"$drive_dir_path\"\n",
        "    ! cp \"$depth_dir_path\"/order_specific.txt \"$drive_dir_path\"\n",
        "\n",
        "    for i in range(1, number_people_max+1):\n",
        "        ! echo ------------------------------------------\n",
        "        ! echo 3d-pose-baseline-vmd [\"$i\"]\n",
        "        ! echo ------------------------------------------\n",
        "\n",
        "        target_name = \"_\" + now_str + \"_idx0\" + str(i)\n",
        "        target_dir = output_json + target_name\n",
        "\n",
        "        !cd ./3d-pose-baseline-vmd && python src/openpose_3dpose_sandbox_vmd.py --camera_frame --residual --batch_norm --dropout 0.5 --max_norm --evaluateActionWise --use_sh --epochs 200 --load 4874200 --gif_fps 30 --verbose 1 --openpose \"$target_dir\" --person_idx 1    \n",
        "\n",
        "        ! echo ------------------------------------------\n",
        "        ! echo VMD-3d-pose-baseline-multi [\"$i\"]\n",
        "        ! echo ------------------------------------------\n",
        "\n",
        "        ! cd ./VMD-3d-pose-baseline-multi && python main.py -v 2 -t \"$target_dir\" -b \"$born_model_csv\" -c 30 -z \"$center_z_scale\" -s \"$smooth_times\" -p \"$threshold_pos\" -r \"$threshold_rot\" -k \"$is_ik\" -e \"$heel_position\" -d \"$depth_smooth_times\"\n",
        "\n",
        "        # INDEX別結果コピー\n",
        "        idx_dir_path = drive_dir_path + \"/idx0\" + str(i)\n",
        "        ! mkdir -p \"$idx_dir_path\"\n",
        "        \n",
        "        # 日本語対策でpythonコピー\n",
        "        for f in glob.glob(target_dir +\"/*.vmd\"):\n",
        "            shutil.copy(f, idx_dir_path)\n",
        "        \n",
        "        ! cp \"$target_dir\"/pos.txt \"$idx_dir_path\"\n",
        "        ! cp \"$target_dir\"/start_frame.txt \"$idx_dir_path\"\n",
        "\n",
        "    # Googleドライブ再マウント\n",
        "    drive.mount('/gdrive')\n",
        "\n",
        "    elapsed_time = (time.time() - start_time) / 60\n",
        "\n",
        "    ! echo \"■■■■■■■■■■■■■■■■■■■■■■■■\"\n",
        "    ! echo \"■■All processing has been completed\"\n",
        "    ! echo \"■■\"\n",
        "    ! echo \"■■Processing time：\" \"$elapsed_time\" \".\"\n",
        "    ! echo \"■■■■■■■■■■■■■■■■■■■■■■■■\"\n",
        "\n",
        "    ! echo \"MMD automatic trace execution result\"\n",
        "    ! echo \"\"\n",
        "\n",
        "    ! echo \"$drive_dir_path\"\n",
        "    ! ls -l \"$drive_dir_path\"\n",
        "\n",
        "from IPython.display import Audio\n",
        "Audio(\"sceneswitch1.mp3\", autoplay=True)        "
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "6mhinYooFuj1",
        "colab_type": "text"
      },
      "source": [
        "## Auto Trace execution (partial execution)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "WKNMqF4LF3zw",
        "colab_type": "text"
      },
      "source": [
        "**If you do not need to execute again after completing「Auto Trace execution (all execution)」, this is the end.** \n",
        "\n",
        "If you want to make any corrections, trace again as follows.\n",
        "\n",
        "1. If you want to change the trace source video\n",
        "  - Upload new videos to **autotrace** folder on Google Drive with **new file name**\n",
        " - Execute the cell of **\"Input video file upload\"**\n",
        " - Execute the cell of **\"Parameter Setting\"**\n",
        " - Execute the cell of **\"Automatic trace execution (all execution)\"**  \n",
        "  \n",
        "2. When you want to change the parameter of【O】\n",
        " - Change the【O】value of **\"Parameter setting\"**\n",
        " - Execute the **\"Parameter Setting\"** cell\n",
        " - Execute the cell of **\"Automatic trace execution (all execution)\"**\n",
        "\n",
        "3. When you want to change the parameter of【F】\n",
        " - Change the【F】value of **\"Parameter setting\"**\n",
        " - Execute the **\"Parameter Setting\"** cell\n",
        " - Execute the cell of **\"A) Automatic trace rerun (depth estimation)\"**\n",
        " - When satisfied with the parameters of【F】\n",
        "       - Execute the cell of **\"B) Auto Trace rerun (2D → 3D)\"**\n",
        "       - Execute the cell of **\"C) Auto Trace rerun (3D → VMD)\"** \n",
        " \n",
        "4. When you want to change the parameter of【V】\n",
        " - Change the【V】value of **\"Parameter setting\"**\n",
        " - Execute the **\"Parameter Setting\"** cell\n",
        " - Execute the cell of **\"C) Auto Trace rerun (3D → VMD)\"** \n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "EQUcMLsf3TcN",
        "colab_type": "text"
      },
      "source": [
        "### A) Auto trace rerun (depth estimation)"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "yM2PNPpd3uai",
        "colab_type": "code",
        "cellView": "form",
        "colab": {}
      },
      "source": [
        "#@markdown Execute this cell after executing the cell for parameter setting.\n",
        "#@markdown It assumes that depth estimation has already been performed in all executions, and performs processing after person index reordering processing.\n",
        "\n",
        "\n",
        "import time\n",
        "import datetime\n",
        "import cv2\n",
        "import os\n",
        "from google.colab import drive\n",
        "\n",
        "start_time = time.time()\n",
        "\n",
        "# 過去深度結果\n",
        "past_depth_dir_path =  output_json + \"_\" + now_str + \"_depth\"\n",
        "\n",
        "# 処理日時\n",
        "now_str = \"{0:%Y%m%d_%H%M%S}\".format(datetime.datetime.now())\n",
        "\n",
        "# Googleドライブマウント\n",
        "drive.mount('/gdrive')\n",
        "\n",
        "# 起点ディレクトリ\n",
        "drive_base_dir = \"/gdrive/My Drive/autotrace\"\n",
        "\n",
        "output_json = \"/content/output/json\"\n",
        "output_openpose_avi = \"/content/output/openpose.avi\"\n",
        "\n",
        "# 出力用Googleドライブフォルダ\n",
        "drive_dir_path = drive_base_dir + \"/\" + now_str \n",
        "! mkdir -p \"$drive_dir_path\"\n",
        "\n",
        "! echo ------------------------------------------\n",
        "! echo mannequinchallenge-vmd\n",
        "! echo ------------------------------------------\n",
        "    \n",
        "# 深度結果コピー\n",
        "depth_dir_path =  output_json + \"_\" + now_str + \"_depth\"\n",
        "\n",
        "! cd mannequinchallenge-vmd && python predict_video.py --video_path \"$input_video\" --json_path \"$output_json\" --interval 20 --reverse_specific \"$reverse_specific\" --order_specific \"$order_specific\" --verbose 1 --now \"$now_str\" --avi_output \"yes\"  --number_people_max \"$number_people_max\" --end_frame_no \"$end_frame_no\" --input single_view --batchSize 1 --past_depth_path \"$past_depth_dir_path\"\n",
        "\n",
        "if os.path.exists( depth_dir_path + \"/error.txt\"):\n",
        "    \n",
        "    # エラー発生\n",
        "    ! cp \"$depth_dir_path\"/error.txt \"$drive_dir_path\"\n",
        "\n",
        "    ! echo \"■■■■■■■■■■■■■■■■■■■■■■■■\"\n",
        "    ! echo \"■■Processing was interrupted due to an error.\"\n",
        "    ! echo \"■■\"\n",
        "    ! echo \"■■■■■■■■■■■■■■■■■■■■■■■■\"\n",
        "\n",
        "    ! echo \"$drive_dir_path\" \"の error.txt の中身を確認してください。\"\n",
        "\n",
        "else:\n",
        "    \n",
        "    ! cp \"$depth_dir_path\"/*.avi \"$drive_dir_path\"\n",
        "    ! cp \"$depth_dir_path\"/message.log \"$drive_dir_path\"\n",
        "    ! cp \"$depth_dir_path\"/reverse_specific.txt \"$drive_dir_path\"\n",
        "    ! cp \"$depth_dir_path\"/order_specific.txt \"$drive_dir_path\"\n",
        "\n",
        "    # Googleドライブ再マウント\n",
        "    drive.mount('/gdrive')\n",
        "\n",
        "    elapsed_time = (time.time() - start_time) / 60\n",
        "\n",
        "    ! echo \"■■■■■■■■■■■■■■■■■■■■■■■■\"\n",
        "    ! echo \"■■[Depth Estimation] processing has been completed\"\n",
        "    ! echo \"■■\"\n",
        "    ! echo \"■■Processing time：\" \"$elapsed_time\" \".\"\n",
        "    ! echo \"■■■■■■■■■■■■■■■■■■■■■■■■\"\n",
        "\n",
        "    ! echo \"\"\n",
        "    ! echo \"MMD automatic trace execution result\"\n",
        "\n",
        "    ! echo \"$drive_dir_path\"    \n",
        "    ! ls -l \"$drive_dir_path\"\n",
        "\n",
        "from IPython.display import Audio\n",
        "Audio(\"sceneswitch1.mp3\", autoplay=True)        "
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Mm0tv4ykzsLn",
        "colab_type": "text"
      },
      "source": [
        "### B) Auto Trace rerun (2D → 3D)"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "MAGaxOsH0Daw",
        "colab_type": "code",
        "cellView": "form",
        "colab": {}
      },
      "source": [
        "#@markdown Execute this cell after executing the cell for parameter setting.\n",
        "#@markdown Execute processing for the number of people.\n",
        "\n",
        "\n",
        "import time\n",
        "import datetime\n",
        "import cv2\n",
        "import os\n",
        "from google.colab import drive\n",
        "\n",
        "start_time = time.time()\n",
        "\n",
        "# Googleドライブマウント\n",
        "drive.mount('/gdrive')\n",
        "\n",
        "# 起点ディレクトリ\n",
        "drive_base_dir = \"/gdrive/My Drive/autotrace\"\n",
        "\n",
        "output_json = \"/content/output/json\"\n",
        "output_openpose_avi = \"/content/output/openpose.avi\"\n",
        "\n",
        "# 出力用Googleドライブフォルダ\n",
        "drive_dir_path = drive_base_dir + \"/\" + now_str \n",
        "! mkdir -p \"$drive_dir_path\"\n",
        "\n",
        "for i in range(1, number_people_max+1):\n",
        "    ! echo ------------------------------------------\n",
        "    ! echo 3d-pose-baseline-vmd [\"$i\"]\n",
        "    ! echo ------------------------------------------\n",
        "\n",
        "    target_name = \"_\" + now_str + \"_idx0\" + str(i)\n",
        "    target_dir = output_json + target_name\n",
        "\n",
        "    !cd ./3d-pose-baseline-vmd && python src/openpose_3dpose_sandbox_vmd.py --camera_frame --residual --batch_norm --dropout 0.5 --max_norm --evaluateActionWise --use_sh --epochs 200 --load 4874200 --gif_fps 30 --verbose 1 --openpose \"$target_dir\" --person_idx 1    \n",
        "\n",
        "    # INDEX別結果コピー\n",
        "    idx_dir_path = drive_dir_path + \"/idx0\" + str(i)\n",
        "    ! mkdir -p \"$idx_dir_path\"\n",
        "    ! cp \"$target_dir\"/pos.txt \"$idx_dir_path\"\n",
        "\n",
        "# Googleドライブ再マウント\n",
        "drive.mount('/gdrive')\n",
        "\n",
        "elapsed_time = (time.time() - start_time) / 60\n",
        "\n",
        "! echo \"■■■■■■■■■■■■■■■■■■■■■■■■\"\n",
        "! echo \"■■[2D → 3D] processing has been completed\"\n",
        "! echo \"■■\"\n",
        "! echo \"■■Processing time：\" \"$elapsed_time\" \".\"\n",
        "! echo \"■■■■■■■■■■■■■■■■■■■■■■■■\"\n",
        "\n",
        "! echo \"\"\n",
        "! echo \"MMD automatic trace execution result\"\n",
        "\n",
        "! echo \"$drive_dir_path\"    \n",
        "! ls -l \"$drive_dir_path\"\n",
        "\n",
        "from IPython.display import Audio\n",
        "Audio(\"sceneswitch1.mp3\", autoplay=True)    "
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "tQh07oDkDCz0",
        "colab_type": "text"
      },
      "source": [
        "### C) Auto Trace rerun (3D → VMD)"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "FULbLzXfyJ2W",
        "colab_type": "code",
        "cellView": "form",
        "colab": {}
      },
      "source": [
        "#@markdown Execute this cell after executing the cell for parameter setting.\n",
        "#@markdown Execute processing for the number of people.\n",
        "\n",
        "import time\n",
        "import datetime\n",
        "import cv2\n",
        "import shutil\n",
        "import glob\n",
        "from google.colab import drive\n",
        "\n",
        "start_time = time.time()\n",
        "\n",
        "# Googleドライブマウント\n",
        "drive.mount('/gdrive')\n",
        "\n",
        "# 起点ディレクトリ\n",
        "drive_base_dir = \"/gdrive/My Drive/autotrace\"\n",
        "\n",
        "output_json = \"/content/output/json\"\n",
        "output_openpose_avi = \"/content/output/openpose.avi\"\n",
        "\n",
        "# 出力用Googleドライブフォルダ\n",
        "drive_dir_path = drive_base_dir + \"/\" + now_str \n",
        "! mkdir -p \"$drive_dir_path\"\n",
        "\n",
        "\n",
        "for i in range(1, number_people_max+1):\n",
        "\n",
        "    ! echo ------------------------------------------\n",
        "    ! echo VMD-3d-pose-baseline-multi [\"$i\"]\n",
        "    ! echo ------------------------------------------\n",
        "    \n",
        "    target_name = \"_\" + now_str + \"_idx0\" + str(i)\n",
        "    target_dir = output_json + target_name\n",
        "\n",
        "    ! cd ./VMD-3d-pose-baseline-multi && python main.py -v 2 -t \"$target_dir\" -b \"$born_model_csv\" -c 30 -z \"$center_z_scale\" -s \"$smooth_times\" -p \"$threshold_pos\" -r \"$threshold_rot\" -k \"$is_ik\" -e \"$heel_position\" -d \"$depth_smooth_times\"\n",
        "\n",
        "    # INDEX別結果コピー\n",
        "    idx_dir_path = drive_dir_path + \"/idx0\" + str(i)\n",
        "    ! mkdir -p \"$idx_dir_path\"\n",
        "    ! cp \"$target_dir\"/*.vmd \"$idx_dir_path\"\n",
        "    # 日本語対策でpythonコピー\n",
        "    for f in glob.glob(target_dir +\"/*.vmd\"):\n",
        "        shutil.copy(f, idx_dir_path)\n",
        "\n",
        "# Googleドライブ再マウント\n",
        "drive.mount('/gdrive')\n",
        "\n",
        "elapsed_time = (time.time() - start_time) / 60\n",
        "\n",
        "! echo \"■■■■■■■■■■■■■■■■■■■■■■■■\"\n",
        "! echo \"■■【3D→VMD】の処理が終了しました\"\n",
        "! echo \"■■\"\n",
        "! echo \"■■処理にかかった時間：\" \"$elapsed_time\" \"分\"\n",
        "! echo \"■■■■■■■■■■■■■■■■■■■■■■■■\"\n",
        "\n",
        "! echo \"\"\n",
        "! echo \"MMD自動トレース実行結果\"\n",
        "\n",
        "! echo \"$drive_dir_path\"\n",
        "! ls -l \"$drive_dir_path\"\n",
        "\n",
        "from IPython.display import Audio\n",
        "Audio(\"sceneswitch1.mp3\", autoplay=True)    "
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "UzsQ7HrjphVF",
        "colab_type": "text"
      },
      "source": [
        "# If an error occurs"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "_JiZw2Ott1Rr",
        "colab_type": "text"
      },
      "source": [
        "If an error occurs, and if the vmd file has not been generated, execute this section one by one from the top.\n",
        "\n",
        "If you still have problems, follow the introductory instructions to share a copy of your notebook."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "1uUbtOC7ptjz",
        "colab_type": "text"
      },
      "source": [
        "## 1. If the number of people in the first frame read by Openpose does not appear"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Mw8zNO_yG-Nt",
        "colab_type": "text"
      },
      "source": [
        "If the error.txt says \"There is no data for the number of people in the first frame\", the reason is that there are no data for the first frame read by Openpose.\n",
        "\n",
        "Please execute the cell below."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ppAVzFJzn4vh",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "!find output/json/ -name \"*.json\" | sort | head -n 1 | xargs ls -l\n",
        "!find output/json/ -name \"*.json\" | sort | head -n 1 | xargs cat"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "4e-MYAS-uLEx",
        "colab_type": "text"
      },
      "source": [
        "【Status】\n",
        "\n",
        "As shown below, if there is no data after people, no human data can be obtained at frame 0.\n",
        "\n",
        "![結果なし](https://drive.google.com/uc?export=view&id=1osssF0NCWply6J0-zPIhN2wm1gT0o6Io)\n",
        "\n",
        "【Solution】\n",
        "\n",
        "Specify the first frame in which a person appears in \"【O】Frame number to start analysis\".\n",
        "\n",
        "The top 30 Openpose result files will be displayed when you execute the cell \"30 list of frames read by Openpose\".\n",
        "\n",
        "![先頭30件](https://drive.google.com/uc?export=view&id=1lxP78w4NIbQSKWhpfbynCjDKOgmcNp0o)\n",
        "\n",
        "JSON files without personal data have very small file size. (In the case of the figure, the 0th frame is no person data)\n",
        "\n",
        "The data size is about 500 bytes (0.5 KB) for one person's data.\n",
        "\n",
        "Please refer to this and decide the top frame number. There is no need to edit or re-upload the original video.\n",
        "\n",
        "When the first frame number is determined, enter the number in \"【O】Frame number to start analysis\" in the \"Parameter setting\" section, and select \"Parameter setting\"> \"Auto Trace execution (all execution)\" in this order. Please practice.\n",
        "\n",
        "\n",
        "In the case of a multi-person trace, it is necessary for all the members to be shown in the 0th frame (【O】Frame number to start analysis).\n",
        "\n",
        "The file size will also increase by the number of people, so please make a guide."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "cYIc5neP9oAl",
        "colab_type": "text"
      },
      "source": [
        "## 2. 30 list of frames read by Openpose"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "qT0qOAp09xlM",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "ls -l output/json/*.json | head -n 30"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "NCckdf_gp0F2",
        "colab_type": "text"
      },
      "source": [
        "## 3. When file is not added to Google Drive"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "wWIauqkWq15H",
        "colab_type": "text"
      },
      "source": [
        "If neither error.txt nor vmd is output, check the cell output.\n",
        "\n",
        "The output itself is successful if there is a final list of file names.\n",
        "\n",
        "However, I have confirmed the case where the data is not reflected even though the linkage with Google Drive has been completed.\n",
        "\n",
        "The details are under investigation, but please download the original data on the cloud for the time being.\n",
        "\n",
        "1. Click the \"File\" column next to the table of contents\n",
        "2. Click \"Update\" in the header\n",
        "3. Open \"output ＞ json\"\n",
        "4. xxx_depth ＞ output_XXX.avi …　Background AVI(MMD)\n",
        "5. xxx_idxXX ＞ output_XXX.vmd　…　Motion data(MMD)\n",
        "6. xxx_idxXX ＞ pos.txt　…　3D joint position data(Unity)\n",
        "7. If you trace multiple people, there are multiple idx.\n",
        "\n",
        "![クラウドデータ](https://drive.google.com/uc?export=view&id=1fArRyRdfs1kBLaLTpdkdJ-MYwHNe-UUq)\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ASco7yEigfFn",
        "colab_type": "text"
      },
      "source": [
        "## 4. If you want to redo everything"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "X0leFRGtgoP7",
        "colab_type": "text"
      },
      "source": [
        "If you have trouble preparing everything and want to start over, reset the runtime.\n",
        "\n",
        "Header >  Runtime > Reset All Runtimes\n",
        "\n",
        "![リセット](https://drive.google.com/uc?export=view&id=1HNOEDju8R5pTZseJ0FCOnFVDRI9ruuLC)\n",
        "\n",
        "A confirmation dialog will appear, so please proceed with OK."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "iOLiarVDw_TY",
        "colab_type": "text"
      },
      "source": [
        "# TIPS"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "oiVbKfOVxByH",
        "colab_type": "text"
      },
      "source": [
        "In addition to the above, I will write things that can be helpful to you as I think."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "RS2M0r4vFH5Z",
        "colab_type": "text"
      },
      "source": [
        "## Recommended work order"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ESNt5KETxMqv",
        "colab_type": "text"
      },
      "source": [
        "I work in the following order.\n",
        "\n",
        "1.  Execute the cell by entering the number of people you want to trace from the trace source video in  ** “【O】Maximum number of people in the video” **\n",
        "2.  Execute the cell of ** \"Auto Trace execution (all execution)\" **\n",
        "3. If the result is an error, execute ** \"Is the number of people in the first frame of Openpose read\" ** in ** \"If an error occurs\" ** and check the frame in which the person appears Do. Move to 6 if successful.\n",
        "4.  Enter the frame number found in step 3 in ** \"【O】Frame number to start analysis\" ** \n",
        "5. Execute the cell of  ** \"Parameter setting\" ** \n",
        "6.  Execute the cell of ** \"A) Auto trace rerun (depth estimation)\" ** \n",
        "7. If the trace can not be recognized in multiple people's trace, specify the order in  ** \"【F】Ordered list\" **  while looking at message.log\n",
        "8.  Execute the cell of ** \"Parameter Setting\" ** \n",
        "9. Execute the cell of ** \"A) Auto trace rerun (depth estimation)\" **\n",
        "10. Repeat 7 to 9 until you are satisfied with the designated order\n",
        "11. If there is an unintended rotation, specify and set the correct inversion status of the corresponding frame in  ** \"【F】Reverse specification list\" **  while looking at message.log\n",
        "\n",
        "  - ※ If the order is changed in the minor lightness people trace, the reverse specified person INDEX will also change, so it is better to execute after the order specification is finished.\n",
        "  \n",
        "12. Execute the cell of  ** \"parameter setting\" **  \n",
        "13.  Execute the cell of ** \"A) Auto trace rerun (depth estimation)\" **\n",
        "14. Repeat 11 to 13 until you are satisfied\n",
        "15. Execute ** \"B) Auto Trace rerun (2D → 3D)\" **   when the specification of replacement or rotation is completed.\n",
        "16.  Adjust the [V] value of “Parameter setting” \n",
        "17. Execute the cell of  ** \"parameter setting\" **  \n",
        "18.  Execute the cell of ** \"C) Auto Trace rerun (3D → VMD)\" **  . Even in the case of a multi-person trace, output for the number of people is performed at one time.\n",
        "19. Repeat 15-18 until you are satisfied\n",
        "\n",
        "good luck!"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "AhrZcOhMxO2f",
        "colab_type": "text"
      },
      "source": [
        "## An easy to trace video"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "assKjr_XxRKL",
        "colab_type": "text"
      },
      "source": [
        " - It is a fixed camera\n",
        " - The joints are clearly visible\n",
        "    - Human body with a hard to see joints such as long skirts and Japanese clothes is not good at analysis\n",
        "    - When the background is a color similar to a person, the shadow is dark, etc., the trace is often mistaken.\n",
        "    - The accuracy is better if the wrist and ankle are seen\n",
        "    - Accuracy is reduced when it is difficult to distinguish between left and right with black trousers etc.\n",
        " - Front facing in the first frame\n",
        "   - I can not get the data of the beginning clearly when I look at the back or side (I often fix it once I look at the front)\n",
        " - The joints of the whole body can be identified in the first frame\n",
        "   - If you are hiding somewhere, the accuracy will drop\n",
        " - Head up, foot down\n",
        "   - If your leg rises high due to a handstand or kick, your foot is misrecognized as a hand. (Especially when the joints at the foot of the foot are above the neck)\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "umaYNzP34XKU",
        "colab_type": "text"
      },
      "source": [
        "## Task"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "SpqJj6mY4ahj",
        "colab_type": "text"
      },
      "source": [
        " - Joint whose rotation can not be estimated\n",
        "  - Wrist\n",
        "  - Finger\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "EtUG-Qsp1QMp",
        "colab_type": "text"
      },
      "source": [
        "# License"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "6OkAoId61RZ7",
        "colab_type": "text"
      },
      "source": [
        "When publishing or distributing the results of \"MMD Auto Trace\", please be sure to confirm the license. The same is true for Unity.\n",
        "\n",
        "I am very grateful if you can enter a license. \n",
        "\n",
        "[MMD Auto Trace License(Japanese)](https://ch.nicovideo.jp/miu200521358/blomaga/ar1686913)"
      ]
    }
  ]
}